Empirical Studies in Multimedia Retrieval Evaluation

The evaluation of retrieval mechanisms for inter-method comparison is necessary in academic as well as in applied research. A major issue in every evaluation is in which way and to what extent the actual perception of the user from the target user group is integrated. Within multimedia retrieval systems the impressions and perceptions of users vary much more than in text retrieval. Empirical studies are a common tool in social science and offer a way to research the correlation between the user perception and the computed similarity between pairs of multimedia documents or a query and the set of results. This approach can be used to complement and extend current evaluation approaches. Within this contribution we summarize general methods from social science and psychology for the interested reader in the area of computer science with some knowledge about statistics. Furthermore we give two examples of undertaken empirical experiments and their outcomes. Within the first one the perception of users is investigated and compared to factors like background and gender, while in the second study metrics are tested upon their ability to reflect the notion of similarity of users. Both experiments aim to give examples and insight on how empirical studies can be used in multimedia research in general and multimedia retrieval evaluation in special.